A Consensus-Based Global Optimization Method with Adaptive Momentum Estimation

Objective functions in large-scale machine-learning and artificial intelligence applications often live high dimensions with strong non-convexity massive local minima. First-order methods, such as the stochastic gradient method Adam, are used to find global Recently, consensus-based optimization (CBO) has been introduced one of gradient-free methods its convergence is proven dimension-dependent parameters, which may suffer from curse dimensionality. By replacing isotropic geometric Brownian motion component-wise one, latest improvement CBO guaranteed converge minimizer dimension-independent although initial data need be well-chosen. In this paper, based on we propose a adaptive momentum estimation (Adam-CBO). Advantages Adam-CBO include: (1) capable finding minima non-convex objective success rates low costs; (2) can handle non-differentiable activation thus approximate low-regularity better accuracy. The former verified by approximating $1000$ dimensional Rastrigin function $100\%$ rate at cost only growing linearly respect latter confirmed solving machine learning task for partial differential equations solutions where provides results than state-of-the-art Adam. A linear stability analysis provided understand asymptotic behavior method.